Method of operating a video gaming system

ABSTRACT

A method of operating a video game system can include applying an optical filter in a field of view of a camera to filter out visible light and allow the camera to view non-visible light directed at a display screen positioned within the field of view of the camera. The method can also include processing the video output feed from the camera with a computer processor to identify a location of a non-visible light dot within the field of view of the camera that is projected onto the display screen by a light targeting peripheral unit. The method can also include calculating the location of the non-visible light dot within the field of view of the camera as a proportion of one or more dimensions of the field of view of the camera and translating the location of the non-visible light dot within the field of view of the camera onto a location on the display screen.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57 andshould be considered a part of this specification.

BACKGROUND Field

The invention is directed to a video gaming system and method ofoperation, and more particularly to a video gaming system using a lighttargeting peripheral.

Description of the Related Art

Video games are very popular. Some games involve targeting objects inthe game. When playing video games, users provide inputs to the gamethrough a controller or peripheral.

SUMMARY

There is a need for an improved video game system.

In accordance with one aspect, a video game system can include aconsole, a camera and a light targeting peripheral (LTP), where the LPTcan direct non-visible light (e.g., infrared (IR) light) onto a displayscreen. The camera can be positioned in a location that allows thecamera to view the display screen, and the camera can locate thenon-visible light dot on the display screen during video game play. Thesystem can be used with any size or type of electronic display (e.g.,any type of television, computer screen, etc.), any type ofnon-electronic display (e.g., projector screen, a wall surface) on whichthe video game is projected, or a display area that does not display avideo game image (e.g., a poster used for calibration), where video gameimages are provided by a separate device (e.g., a virtual realitydevice, such as a head mounted display worn by the user), for example.

In accordance with another aspect, a video game system is provided. Thesystem comprises a console configured to transmit one or more visual oraural effects of a video game to a display screen. The system alsocomprises a camera positionable so that it can view the display screen,the camera configured to communicate with the console. The system alsocomprises a light targeting peripheral comprising a trigger and anon-visible light source, the light targeting peripheral configured todirect a non-visible light onto the display screen and to communicatewith one or both of the console and the camera when the trigger isactuated, the camera configured to locate a position on the displayscreen targeted by the light targeting peripheral upon the actuation ofthe trigger irrespective of a size and type of the display screen and tocommunicate said position to the console.

In accordance with another aspect, a video game system is provided. Thesystem comprises a console configured to transmit one or more visual oraural effects of a video game and a camera positionable so that it canview a display, the camera configured to communicate with the console.The system also comprises a light targeting peripheral comprising atrigger and a non-visible light source, the light targeting peripheralconfigured to direct a non-visible light onto the display and tocommunicate with one or both of the console and the camera when thetrigger is actuated, the camera configured to locate a position on thedisplay targeted by the light targeting peripheral upon the actuation ofthe trigger irrespective of a size and type of the display screen and tocommunicate said position to the console.

In accordance with another aspect, a video game system is provided. Thesystem comprises a camera positionable so that it can view a display.The camera comprises an optical filter operable to filter out visiblelight while allowing non-visible light to pass through a lens of thecamera during operation of a video game. The system also comprises alight targeting peripheral comprising a trigger and a non-visible lightsource, the light targeting peripheral configured to direct anon-visible light onto the display and to communicate with the camerawhen the trigger is actuated, the camera configured to locate a positionon the display targeted by the light targeting peripheral upon theactuation of the trigger irrespective of a size and type of the display.

In accordance with another aspect, a method of operating a video gamesystem is provided. The method comprises applying an optical filter in afield of view of a camera to filter out visible light and allow thecamera to view non-visible light directed at a display screen positionedwithin the field of view of the camera. The method also comprisesprocessing the video output feed from the camera with a computerprocessor to identify a location of a non-visible light dot within thefield of view of the camera that is projected onto the display screen bya light targeting peripheral unit. The method also comprises calculatingthe location of the non-visible light dot within the field of view ofthe camera as a proportion of one or more dimensions of the field ofview of the camera. The method further comprises translating thelocation of the non-visible light dot within the field of view of thecamera onto a location on the display screen.

In accordance with another aspect, a method of operating a video gamesystem is provided. The method comprises mechanically positioning anoptical filter in a field of view of a camera to filter out visiblelight and allow the camera to view non-visible light directed at adisplay screen positioned within the field of view of the camera. Themethod also comprises searching the video output feed from the camerawith a computer processor to identify a location within the field ofview of the camera where a non-visible light dot is present or absentupon actuation of a trigger of a light targeting peripheral thatprojects non-visible light onto the display screen. The method alsocomprises calculating the location within the field of view of thecamera where the non-visible light dot is present or missing as aproportion of dimensions of the field of view of the camera. The methodfurther comprises translating the location of the presence or absence ofthe non-visible light dot within the field of view of the camera onto alocation on the display screen by applying the proportion of saiddimensions of the field of view of the camera to correspondingdimensions on the display screen.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic drawing of one embodiment of a video gamingsystem.

FIG. 2A shows one embodiment of a camera for the system in FIG. 1, withan optical filter positioned in front of the lens to filter out visiblelight and allow non-visible light to pass into the camera lens.

FIG. 2B shows the camera of FIG. 2A, with an optical filter positionedout of the way to allow visible light into the camera lens.

FIG. 2C shows a schematic view of the electronics of the camera of FIG.1.

FIG. 3 shows a flowchart for one method of calibrating the camera.

FIG. 3A shows one embodiment of a calibration image.

FIG. 3B shows a calibration step in the method of FIG. 3.

FIG. 3C shows a calibration step in the method of FIG. 3.

FIG. 3D shows another embodiment of a calibration step in the method ofFIG. 3.

FIG. 3E shows another embodiment of a calibration step in the method ofFIG. 3.

FIG. 4 shows a flowchart for one method of recalibrating the camera.

FIG. 5 shows a flowchart of one embodiment of a tracking method for thevideo gaming system.

FIG. 5A shows a schematic drawing of the video gaming system in use.

FIG. 5B shows a step in the tracking method of FIG. 5.

FIG. 5C shows another embodiment of a step in the tracking method ofFIG. 5.

FIG. 5D shows another embodiment of a step in the tracking method ofFIG. 5.

FIG. 5E shows a step in the tracking method of FIG. 5.

FIG. 5F shows a step in the tracking method of FIG. 5.

FIG. 6 is a schematic block diagram of the electronics of a lighttargeting peripheral used with the video gaming system of FIG. 1.

FIG. 7 is a schematic drawing of the video gaming system in use by twoindividuals during use.

DETAILED DESCRIPTION

FIG. 1 shows one embodiment of a video gaming system 100. The system 100can include a console 10, a light locating camera 20 that utilizes anoptical filter 30, and a light targeting peripheral (LTP) 40 (e.g., alight gun). Advantageously, the system 100 can be used with any size ortype of display screen, including any type of television (e.g., LCD,LED, CRT, Plasma, etc.), computer monitor, display area (e.g., aprojector screen, a wall area) that can display a video game image(e.g., projected onto the display area by a projector), or a displayarea that does not display a video game image (e.g., a poster used forcalibration), where video game images are provided by a separate device(e.g., a virtual reality device, such as a head mounted display worn bythe user), for example.

The LTP 40 can direct non-visible light (e.g., an infrared (IR) light,ultraviolet (UV) light, etc.) onto a display screen 50, and the locationof the light on the display screen 50 can be located by the camera 20,as discussed in more detail below. The console 10 can communicate withthe display screen 50 (e.g., via a cable 12), and the camera 20 cancommunicate with the console 10 (e.g., via a cable 14). The console 10can operate the video game and communicate with the display screen 50 todisplay images from the video game and provide audio effects associatedwith the video game. In some embodiments, one or more of the console 10,camera 20 and LTP 40 can communicate wirelessly (e.g., via an RF link).

In the illustrated embodiment, the LTP 40 can communicate wirelesslywith the camera 20, such as via an RF link. In another embodiment, theLTP 40 can additionally, or alternatively, communicates with the console10. For example, if the image processing is at least partially performedby software in the console 10, the LTP 40 can communicate with theconsole 10. The LTP 40 can have a wireless transmitter 42 to transmitdata to one or both of the console 10 and the camera 20, which can havea wireless receiver 22, as discussed further below. Optionally, the LTP40 can also have a wireless receiver to receive data from one or both ofthe console 10 and the camera 20. In still another embodiment, the LTP40 can communicate with one or both of the camera 20 and console 10 viaa cable. The LTP 40 can have a light source 44, which in one embodimentcan be an infrared laser, where the IR laser can have optics tocollimate and focus the laser so that the laser is straight, and wherethe size of the IR light dot from the laser has the same size regardlessof the distance of the LTP 40 from the display screen 50. In anotherembodiment, the LTP 40 can have an infrared LED as the light source 44,with optics to focus the IR light from the LED. The size of the IR lightdot will vary with the distance of the LTP 40 is from the display screen50 (e.g., the IR light dot will be larger the farther the LTP 40 is fromthe display screen 50). However, in other embodiments, the light source44 of the LTP 40 can use other forms of non-visible light (e.g.,ultraviolet light).

The LTP 40 can optionally be customizable. In one embodiment, the LTP 40can be a passive device and include a non-visible light emitter (e.g.,an IR light emitter) as the light source 44 and a trigger 46. In oneembodiment, actuation of the trigger 46 can break or switch off thenon-visible light that is otherwise continuously emitted. In anotherembodiment, actuation of the trigger 46 does not break or switch off thenon-visible light but communicates with the camera 20 and/or console 10when the trigger 46 is actuated. In still another embodiment, theactuation of the trigger 46 can turn on the non-visible lightmomentarily, where the non-visible light is otherwise switched off, asdescribed further below. If the LTP 40 is a wireless device, it can alsoinclude one or more batteries (not shown) and the wireless transmitter42. Where the LTP 40 is a passive device, it does not receive any datafrom the camera 20.

In another embodiment, the LTP 40 can be an active device. In thisembodiment, in addition to having the features described above for apassive device, the LTP 40 can include a receiver (e.g. wirelessreceiver) and circuitry (e.g., integrated circuit or IC) that allows itto have additional functionality. For example, the LTP 40 can receivedata from the camera 20 that allows it to control aspects of the videogame itself.

In another embodiment, the LTP 40 can include additional control inputsin addition to the trigger 46 that allow the LTP 40 to provideadditional inputs to the video game, such as when the LTP 40 is anactive device as described above. For example, the LTP 40 can includeone or more of a D-pad, one or more input buttons, and an analogjoystick.

In another embodiment, the LTP 40 can have one or more sensors, such asaccelerometers, magnetometers, gyroscopes, that can be used in inertialtracking of the LTP 40 by tracking software of the system 100. Saidinertial tracking can facilitate the determination of the location ofthe non-visible light dot when the trigger is actuated (e.g., where thenon-visible light is continually emitted and therefore continuallytracked by the tracking software) by providing the tracking softwarewith additional information about the direction of movement and/ororientation of the LTP 40 before the trigger 46 was actuated.

In another embodiment, the LTP 40 can have one or more expansion portsthat can receive a controller to provide additional control in the videogame. In such an embodiment, the LTP 40 can operate in tandem with thecontroller to provide a user with the ability to control more aspects ofthe video game than if the controller was not coupled to the LTP 40.

The camera 20 can be positioned in any location that allows it to viewor “look” at the display screen 50 (e.g., at a location below, above, tothe left of, to the right of, close to, or far from the display screen50). Moreover, the camera 20 can be supported on any suitable surface,such as the floor, on a table, a counter, etc. The camera 20 can bemounted in any suitable manner. In one embodiment, the camera 20 canhave a clamp that allows it to mount, for example, to a table (e.g.,coffee table) or chair. In another embodiment, the camera 20 can havevacuum mount system (e.g., one or more vacuum cups or suction cups) thatcan removably mount the camera 20 to a support surface (such as a tablesurface, floor, etc.) to generally fix the location of the camera 20. Instill another embodiment, the camera 20 can have one or more pads(friction pads) on a bottom surface of the camera 20 to generally fixthe location of the camera 20. In one embodiment, the camera 20 can havea low profile that inhibits unintended movement of the camera 20, forexample, when bumped by a user. Optionally, the camera 20 can be domeshaped. In one embodiment, the camera 20 can have an accelerometer orother type of orientation sensor (e.g., magnetometer) that can provideinformation on the orientation of the camera 20, which can be used bythe software (e.g., calibration software or tracking software, asdescribed further below) to ensure that the location of a non-visiblelight when the trigger 46 is actuated is correctly translated onto thedisplay screen 50.

With continued reference to FIG. 1, the camera 20 can have an opticalfilter 30 that filters out visible light but allows the camera 20 toview non-visible light, such as a low-pass IR optical filter, where thefilter 30 is selectively applied to filter out visible light. In oneembodiment, the optical filter 30 can resemble a piece of black plastic.FIG. 2A shows an embodiment of the camera 20 with the optical filter 30positioned in front of the lens 23 of the camera 20 to filter outvisible light while allowing non-visible light (e.g., IR light) to passthrough the lens 23 of the camera 20. FIG. 2B shows the camera 20 ofFIG. 2A, where the optical filter 30 is positioned out of the way of thelens 23 to allow visible light to pass through the lens 23 of the camera20.

In one embodiment, the filter 30 is a switchable or removable filter 30that can be selectively positioned in front of the lens 23 of the camera20. For example, in one embodiment, the optical filter 30 can bepositioned mechanically in front of the lens 23 of the camera 20 andmoved out of the way from in front of the lens 23 of the camera 20, suchas with a stepper motor. However, other suitable mechanisms can be usedto position the optical filter 30 in front of the lens 23 of the camera20 as well as move it out of the way of the lens 23 of the camera 20,such as a sliding mechanism. In still another embodiment, the opticalfilter 30 can be manually positioned in front of the lens 23 of thecamera 20 (e.g., by a user) as well as removed from in front of the lens23. In another embodiment, the camera 20 can have an electronicallyswitchable optical filter (e.g., night mode), as opposed to a physicaloptical filter, that can be selectively operated to filter out visiblelight while allowing non-visible light (e.g., IR light) to pass throughthe lens 23 of the camera 20. In some embodiments, the camera 20 can bea wide bandwidth camera, where the IR blocking filter has been removedso that the camera 20 can view IR through visible light. In oneembodiment, the camera 20 is a Raspberry Pi Noir IR camera.

FIG. 2C shows one embodiment of an electronics diagram for the camera20. The camera 20 can include an optical filter 30 and an actuator 32that selectively positions the filter 30 in front of the lens 23 tofilter out visible light. Optionally, the light that passes through thelens 23 can be processed by an image processing circuit 24 in the camera20. In another embodiment, the camera 20 can exclude the imageprocessing circuit and such circuit can be in the console 10. The camera20 can also include a communication module, which can be connected tothe wireless receiver 22 and the cable 14 that connects the camera 20 tothe console 10. The camera 20 can also have control circuitry 27 forcontrolling the operation of the camera 20, which can optionally includeone or more sensors (e.g., accelerometer, magnetometer) and a memory 28.In one embodiment, the camera 20 received power from an outside sourcevia a power module 29 and cable. In another embodiment, the camera 20can be powered by one or more batteries (not shown).

Calibration

Prior to use of the system 100, the position of the camera 20 iscalibrated relative to the display screen 50. FIG. 3 shows a flowchartof a calibration method 200 for the camera 20. The optical filter 30 isremoved 210 from the camera 20 (or the optical filter is switched offwhere the camera 20 has an electronically switchable optical filter) sothat the camera 20 can see visible light. A calibration image isdisplayed 230 on the display screen 50 (e.g., by the console 10). Imagerecognition software processes 250 the visible light video feed (e.g.,continually reads the video output) from the camera 20 to find thecalibration image. In one embodiment the image processing is at leastpartially performed by the camera 20 (via the image processing circuit24, as described above). In another embodiment, the image processing isat least partially performed by the console 10. Once the calibrationimage is found by the software, the software calculates 270 cameracoordinates that represent the corners of the display screen 50 in thefield of view of the camera 20, at which point the calibration of thecamera 20 is completed. In some embodiments, where the camera 20 has anextra wide angle lens that distorts the image and where the displayscreen 50 appears curved, the image recognition software utilizesadditional algorithms to undistort the image.

With respect to step 230, various different images can be used in thecalibration method 200. In one embodiment, the calibration image can bea QR code. In another embodiment, the calibration image can be a plainscreen image that cycles through a known sequence of colors on thedisplay screen 50 (e.g., all red, all blue, all green, etc.), where theimage recognition software can process the image and look for pixels inthe live feed from the camera 20 that mimics the sequence of thecalibration image.

FIG. 3A shows one embodiment of a calibration image 230A that can beused in step 230 of the calibration method 200, where the calibrationimage 230A includes nine markers 232 that are displayed on the displayscreen 50. The image recognition software can process 250 thecalibration image using a Harris Corner Detection algorithm. In anotherembodiment, the image recognition software can process 250 thecalibration image using other suitable algorithms, such as a Featuresfrom Accelerated Segment Test (FAST) algorithm. Further details on theFAST algorithm can be found in the following articles, all of which areincorporated by reference: Edward Rosten et al., Fusing Points and Linesfor High Performance Tracking, IEEE International Conference on ComputerVision, October 2005; Edward Rosten et al., Machine Learning forHigh-speed Corner Detection, European Conference on Computer Vision, May2006; and Edward Rosten et al., FASTER and Better: A Machine LearningApproach to Corner Detection, IEEE Trans. Pattern Analysis and MachineIntelligence, 2010.

The Harris Corner Detection algorithm returns a numerical value for eachpixel in the camera image representative of a likelihood of the pixelbeing in a corner in the image. The algorithm filters out any pixelbelow a threshold value and generates a list of points (x, ycoordinates) within the camera image that represent potential corners.The list of points are processed and grouped into clusters. For example,each point is processed to determine if it is within a distancethreshold from any other point previously processed within an existingcluster; if the point is within the threshold it is added to thecluster. If said point is not within the threshold from another point, anew cluster is started with said point. This process is repeated untilall points are processed. The center of each cluster is then calculated,which can be defined as the average coordinate of all the points in thecluster, and the radius of each cluster is also calculated, which is thefurthest distance any of the points in the cluster is from the center ofthe cluster. Clusters having less than a predetermined number of points(e.g., less than 2 points) are rejected or filtered out.

FIG. 3B shows a sub-step 230B of the processing step 250 of thecalibration process 200 with the calibration image on the display screen50, where all the detected clusters are represented by a circle and anidentification number. As shown in FIG. 3B, the software has detectedmost of the reference marks, but there may be a few that aremisidentified (e.g., no. 7 in FIG. 3B).

FIG. 3C shows a sub-step 230C of the processing step 250 in thecalibration process 200 with the calibration image on the display screen50. The image recognition software determines if the clusters representa screen by using the known relationship of the reference marks in thecalibration image to each other. Each cluster in the list of clusters isevaluated as an “assumed” center point of the display screen 50 bydetermining if it meets the following set of rules. The imagerecognition software looks for pairs of clusters that can be connectedby a line 234 that passes in close proximity to the center point of the“assumed” center point, and where the end points of such a line areapproximately at the center of another line defined by another pair ofclusters. In FIG. 3C, the line from cluster 1 to cluster 7 passesthrough the “assumed” center point (at cluster 4) and the ends of saidline are approximately at the center of the lines that connect clusters0 to 2 and clusters 6 to 8. Similarly, the line 234 that passes fromcluster 3 to 5 passes through the “assumed” center point (at cluster 4)and the ends of said line are approximately at the center of the linesthat connect clusters 2 to 8 and that connect clusters 0 to 6. The endpoint lines are evaluated to ensure they form a closed rectangle aroundthe center point, and diagonal lines from the intersection of the fourend point lines are evaluated to confirm the lines pass through the“assumed” center point. Additionally, angles between the four lines(e.g., between the line from cluster 0 to 2, and line from cluster 2 to8) are evaluated to determine if the angle meets a threshold value (e.g.around 75 degrees). Because the camera 20 may be askew to the displayscreen 50, the angle between the lines may not be approximately 90degrees. Once the image recognition software has confirmed the “assumed”center point of the display screen 50 is the center point of the displayscreen 50, the software determines 270 the coordinates of the fourcorners of the display screen 50 within the field of view of the camera20 and the calibration is complete (e.g., P1-P4 in FIG. 5B).

In another embodiment of the calibration method 200, the calibrationimage, such as the calibration image 230A in FIG. 3A, can be flashed onin one frame on the display screen 50 (e.g., by the console 10) andturned off in the next frame on the display screen 50. For example, inone embodiment the calibration image 230A can be flashed on for 100 msand turned off for 100 ms. However, in other embodiments, the time thecalibration image 230A is on and off can vary (e.g., can be about 150ms). In the illustrated embodiment, the calibration image 230A has ninemarkers. However, in other embodiments the calibration image 230A canhave a different number of markers (e.g., fewer markers, more markers).In one embodiment, the calibration image 230A can have only four markersat the corners.

Image recognition software can process 250 the feed from the camera 20frame by frame and perform frame subtraction to identify the changes inwhat is shown in the display screen 50 (e.g., identify changes on thedisplay screen when the calibration image 230A is turned on and off). Inone embodiment, the camera 20 can operate at about 90 frames per second;however, in other embodiments, the camera 20 can operate at a larger orsmaller number of frames per second. Where the flashed frames of thecalibration image 23A are not synchronized with the camera frames, theimage processing software can use an algorithm, including an accumulatoror counter, to account for the calibration image flashing at a slowerrate than the camera operation, in order to perform the framesubtraction step and identify the calibration image. In one embodiment,the software compares a camera frame with a previous frame (e.g., aframe from 10-15 previous frames) that is not the immediately previousframe. In another embodiment, the software compares a camera frame withthe immediately previous frame.

Due to the flashing of the calibration image 230A on the display screen50, the frame subtraction process results in the background detail beingremoved (e.g., background detail that may be interpreted as falsecorners) and leaving only the calibration markers in a black screen (notshown), as shown in FIG. 3D. In some embodiments, the image calibrationsoftware can look for the flashing calibration image over a number offrames to identify the markers in the calibration image as reliablemarkers. The image recognition software can then associate the markerswith the corners of the screen and the software determines 270 thecoordinates of the four corners on the display screen 50 within thefield of view of the camera 20 (e.g., connects the markers with lines todefine the corners of the display screen 50 within the field of view ofthe camera 20), as shown in FIG. 3E. The flashing of the calibrationimage and use of frame subtraction can advantageously simplify andspeed-up the calibration process.

In one embodiment, the calibration method 200 can be performed everytime the system 100 is turned on, prior to the user playing a game. Inanother embodiment, the calibration method 200 can be performed at apredetermined time (e.g., once a day, once every other day, once everyweek, etc.).

Recalibration

FIG. 4 shows one embodiment of a recalibration method 300, which can beperformed after the calibration method 200 has been completed (e.g.,while in play mode, discussed further below). If the camera 20 positionmoves (e.g., as a result of a player bumping into the camera), thecamera 20 can be recalibrated relative to the display screen 50. Withcontinued reference to FIG. 4, if motion of the camera 20 is sensed 310(e.g., with one or more sensors in the camera 20, such as anaccelerometer), the software determines 320 if the sensed motion isabove a predetermined threshold (e.g., above a predetermined forcemagnitude). In one embodiment, the one or more sensors in the camera 20that sense whether the camera 20 has been moved can be the same sensorsused to determine the orientation of the camera 20, as discussed above;in another embodiment, the sensors that sense movement of the camera 20to determine if recalibration should be performed can be separate fromthe one or more sensors that sense the orientation of the camera 20. Ifthe sensed movement is above the threshold, then the calibration method200 is performed again (see FIG. 3). In one embodiment, if the sensedmotion is above the predetermined threshold, the system 100 mayautomatically being the recalibration of the camera 20. In anotherembodiment, the system 100 can provide feedback to the user (e.g.,visual and/or audio feedback) that the camera 20 needs to berecalibrated, at which point the video game is paused and the camera 20is recalibrated. If the sensed motion 310 is not above the threshold,the software determines that recalibration is not needed.

Play Mode—Tracking Non-Visible Light

FIG. 5 shows a flowchart of a method of tracking the non-visible light(e.g., IR light) from the LTP 40 when the system 100 is in play mode(shown in FIG. 5A). Once calibration of the camera 20 is completed, theoptical filter (such as optical filter 30) is positioned 410 in front ofthe lens 23 of the camera 20 to filter out visible light while allowingnon-visible light to pass through the lens 23 of the camera 20.

The tracking software searches the feed from the camera 20 fornon-visible light (e.g., IR light). As shown in FIG. 5B, there may bemultiple sources of non-visible light, such as background IR light on awall IR2, IR light on an LED TV IR3, IR reflection on the display screen50 IR4, so the software determines 430 which source is most likely to bethe non-visible light IR1 from the LTP 40. In one embodiment, thesoftware determines which source in the video feed is the non-visiblelight IR1 from the LTP 40 by filtering out intensity values below apredetermined threshold value (e.g., to filter out background light). Inanother embodiment, the software additionally, or alternatively, ignores(filters out) light sources that aren't round (or less dense, asdiscussed further below). For example, the software searches for a groupof pixels that form a dot (e.g., generally circular shape) with thehighest concentration or density of pixels, while considering theintensity values. In still another embodiment, the software ignores(filters out) light sources that are outside the quadrangle defined bythe corners of the display screen (P1-P4 in FIG. 5B) that are in thefield of view of the camera 20. However, other suitable techniques foridentifying the non-visible light IR1 from the LTP 40 can be used.

In another embodiment, the image processing software compares a frame(see FIG. 5C), in which the non-visible light IR1 from the LTP 40 hasnot been actuated, with the camera frame in FIG. 5B, in which thenon-visible light IR1 from the LTP 40 has been actuated, and uses aframe subtraction algorithm, as discussed above, to filter out sourcesof non-visible light that are not associated with the LTP 40, resultingin the image on FIG. 5D, where non-visible light from other than the LTP40 has been filtered out. In some embodiments, the image processingsoftware can further use an algorithm to confirm that the remainingnon-visible light on the screen 50 corresponds to the light from the LTP40. For example, as discussed above, the software can search for a groupof pixels that form a dot (e.g., generally circular shape) with thehighest concentration or density of pixels, while considering theintensity values.

Once the location of the non-visible light IR1 from the LTP 40 isdetected (e.g., at location t_(x), t_(y) in FIG. 5B or FIG. 5D), thetracking software expresses 450 the position of the non-visible lightdot IR1 as a proportion of the quadrangle (P1-P4). In one embodiment,shown in FIG. 5E, this is done by projecting a line from one of thecorners (e.g., P1) of the quadrangle through the detected location(t_(x), t_(y)) of the non-visible light dot IR1 (e.g., IR light dot),calculating where it intersects the opposite side of the quadrangle(P1-P4), and expressing said intersection on the opposite side of thequadrangle (P1-P4) as a proportion of the distance L2 along saidopposite side (e.g., Y1/Y2); this process of projecting a line from acorner through the detected location of the non-visible light dot andcalculating the intersection of the line on an opposite side of thequadrangle and expressing it as a proportional distance along saidopposite side can be repeated for one or more other corners of thequadrangle (P1-P4).

As shown in FIG. 5F, the tracking software then maps 470 saidproportional distances (e.g., L2*Y1/Y2, L1*X1/X2 in FIG. 5E) on thesides of the quadrangle (P1-P4) onto the display screen 50 to identifyintersection points along corresponding borders of the display screen 50(e.g., X1/X2 and Y1/Y2 in FIG. 5E is equal to X3/X4 and Y3/Y4 in FIG.5F, respectively). Lines are then calculated 490 from said intersectionpoints to one or more corners (e.g., P1, P3) of the display screen 50opposite the intersection location, and the intersection point of allthe lines corresponds to the location t_(rx), t_(ry)) of the non-visiblelight dot (e.g., IR light dot) on the display screen 50.

In the tracking process described above, only two lines need to be drawnfrom corners of the quadrangle (P1-P4) through the detected location(t_(x), t_(y)) of the non-visible light dot and onto opposite sides ofthe quadrangle, which can then be translated to the display screen 50 asdiscussed above to translate the camera view to the display screen 50 or“game space”. However, use of four lines (one from each of the cornersof the quadrangle) that intersect the detected non-visible light dotlocation can improve the accuracy of providing the non-visible light dotlocation on the display screen 50 and provides for error redundancy.

In one embodiment, the non-visible light (e.g., IR light) is continuallyprojected by the LTP 40, and the software is constantly tracking thenon-visible light dot, but can in one embodiment only record itsposition once the trigger 46 on the LTP 40 is actuated. In anotherembodiment, the non-visible light is continually projected by the LTP40, but the light is switched off when the trigger 46 is actuated; inthis embodiment, the tracking software searches the feed from the camera20 for the location where the non-visible light dot is missing, andtranslates this location to the display screen 50, as discussed above.Advantageously, this method of continually projecting the non-visiblelight and recording only when the trigger 46 is actuated provides arelatively fast system response (e.g., minimum latency) from theactuation of the trigger 46 to the video game providing feedback to theuser (e.g., visual and/or audio feedback). However, the location of thenon-visible light dot that is recorded corresponds to a location wherethe LTP 40 was pointed a fraction of a second prior, and one or morebatteries of the LTP 40 may drain more quickly (e.g., where the LTP 40is wireless) since the non-visible light would be constantly projected.

In another embodiment, the non-visible light is only projected by theLTP 40 when the trigger 46 is actuated. The software then searches thefeed from the camera 20 to identify the non-visible light dot, asdiscussed above. Advantageously, this method results in increasedaccuracy of the actual position when the trigger 46 is actuated, and theone or more batteries of the LTP 40 (e.g., where the LTP 40 is wireless)may drain more slowly since the light is only projected when the trigger46 is actuated. However, due to the latency in reviewing the feed fromthe camera 20, it may take several milliseconds for the non-visiblelight dot to be identified in the camera feed and communicated to theconsole 10, which may increase the latency for the user to receivefeedback (e.g., visual and/or audio feedback) following the actuation ofthe trigger 46.

FIG. 6 shows a schematic diagram of the electronics in the LTP 40. TheLTP 40 can have a light source or emitter 44, as discussed above, whichcan be electrically connected to control circuitry 41. The controlcircuitry 41 can be electrically connected to a communication module 43,which includes a wireless transmitter 42, as discussed above. The LTP 40can also have a switch 47 that is actuated by the trigger 46 and thatcommunicates with the control circuitry 41 to indicate when the trigger46 has been actuated. The LTP 40 can also have one or more batteries 48that can power the electronics, including the control circuitry 41,communication module 43 and switch 47. In one embodiment, as discussedabove, the LTP 40 can have one or more sensors, which can be part of thecontrol circuitry 41.

Though FIG. 1 shows a single LTP 40, one of skill in the art willrecognize that more than one LTP 40 can be utilized while playing thevideo game (e.g., two LTPs 40, four LTPs 40), depending on the number ofplayers, as shown in FIG. 7. As discussed above, the LTP 40 cancommunicate with one or both of the camera 20 and console 10 via an RFlink. To accommodate multiple users during a video game, the system 100can utilize a time division multiplexing method so that the multipleLTPs 40 are not all allowed to be actuated at the same time.

The console 10 can optionally have one or more processors, acommunication interface, a main memory, a ROM and a storage device,which can be connected to a bus.

While certain embodiments of the inventions have been described, theseembodiments have been presented by way of example only, and are notintended to limit the scope of the disclosure. Indeed, the novel methodsand systems described herein may be embodied in a variety of otherforms. Furthermore, various omissions, substitutions and changes in thesystems and methods described herein may be made without departing fromthe spirit of the disclosure. The accompanying claims and theirequivalents are intended to cover such forms or modifications as wouldfall within the scope and spirit of the disclosure. Accordingly, thescope of the present inventions is defined only by reference to theappended claims.

Features, materials, characteristics, or groups described in conjunctionwith a particular aspect, embodiment, or example are to be understood tobe applicable to any other aspect, embodiment or example described inthis section or elsewhere in this specification unless incompatibletherewith. All of the features disclosed in this specification(including any accompanying claims, abstract and drawings), and/or allof the steps of any method or process so disclosed, may be combined inany combination, except combinations where at least some of suchfeatures and/or steps are mutually exclusive. The protection is notrestricted to the details of any foregoing embodiments. The protectionextends to any novel one, or any novel combination, of the featuresdisclosed in this specification (including any accompanying claims,abstract and drawings), or to any novel one, or any novel combination,of the steps of any method or process so disclosed.

Furthermore, certain features that are described in this disclosure inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations, one or more features from a claimedcombination can, in some cases, be excised from the combination, and thecombination may be claimed as a subcombination or variation of a subcombination.

Moreover, while operations may be depicted in the drawings or describedin the specification in a particular order, such operations need not beperformed in the particular order shown or in sequential order, or thatall operations be performed, to achieve desirable results. Otheroperations that are not depicted or described can be incorporated in theexample methods and processes. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the described operations. Further, the operations may berearranged or reordered in other implementations. Those skilled in theart will appreciate that in some embodiments, the actual steps taken inthe processes illustrated and/or disclosed may differ from those shownin the figures. Depending on the embodiment, certain of the stepsdescribed above may be removed, others may be added. Furthermore, thefeatures and attributes of the specific embodiments disclosed above maybe combined in different ways to form additional embodiments, all ofwhich fall within the scope of the present disclosure. Also, theseparation of various system components in the implementations describedabove should not be understood as requiring such separation in allimplementations, and it should be understood that the describedcomponents and systems can generally be integrated together in a singleproduct or packaged into multiple products.

For purposes of this disclosure, certain aspects, advantages, and novelfeatures are described herein. Not necessarily all such advantages maybe achieved in accordance with any particular embodiment. Thus, forexample, those skilled in the art will recognize that the disclosure maybe embodied or carried out in a manner that achieves one advantage or agroup of advantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

Conditional language, such as “can,” “could,” “might,” or “may,” unlessspecifically stated otherwise, or otherwise understood within thecontext as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements, and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements, and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements, and/or steps areincluded or are to be performed in any particular embodiment.

Conjunctive language such as the phrase “at least one of X, Y, and Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z. Thus, such conjunctive language is not generallyintended to imply that certain embodiments require the presence of atleast one of X, at least one of Y, and at least one of Z.

Language of degree used herein, such as the terms “approximately,”“about,” “generally,” and “substantially” as used herein represent avalue, amount, or characteristic close to the stated value, amount, orcharacteristic that still performs a desired function or achieves adesired result. For example, the terms “approximately”, “about”,“generally,” and “substantially” may refer to an amount that is withinless than 10% of, within less than 5% of, within less than 1% of, withinless than 0.1% of, and within less than 0.01% of the stated amount. Asanother example, in certain embodiments, the terms “generally parallel”and “substantially parallel” refer to a value, amount, or characteristicthat departs from exactly parallel by less than or equal to 15 degrees,10 degrees, 5 degrees, 3 degrees, 1 degree, or 0.1 degree.

The scope of the present disclosure is not intended to be limited by thespecific disclosures of preferred embodiments in this section orelsewhere in this specification, and may be defined by claims aspresented in this section or elsewhere in this specification or aspresented in the future. The language of the claims is to be interpretedbroadly based on the language employed in the claims and not limited tothe examples described in the present specification or during theprosecution of the application, which examples are to be construed asnon-exclusive.

What is claimed is:
 1. A method of operating a video game system,comprising: applying an optical filter in a field of view of a camera tofilter out visible light and allow the camera to view non-visible lightdirected at a display screen positioned within the field of view of thecamera; processing a digital video output feed from the camera with acomputer processor to identify a location of a non-visible light dotwithin the field of view of the camera that is projected onto thedisplay screen by a light targeting peripheral unit; calculating thelocation of the non-visible light dot within the field of view of thecamera as a proportion of one or more dimensions of the field of view ofthe camera; translating the location of the non-visible light dot withinthe field of view of the camera onto a location on the display screen;and initiating a recalibration of the camera to recalculate a boundaryof the display screen in the field of view of the camera if a motion ofthe camera above a threshold amount is sensed with a motion sensor. 2.The method of claim 1, wherein applying the optical filter includesmechanically positioning the optical filter in the field of view of thecamera.
 3. The method of claim 1, wherein applying the optical filterincludes turning on an electronically switchable optical filter.
 4. Themethod of claim 1, wherein processing the digital video feed from thecamera includes continuously processing the video output feed.
 5. Themethod of claim 1, wherein said processing is performed at least in partby the camera.
 6. The method of claim 1, wherein said processing isperformed at least in part by a console of the video game system.
 7. Themethod of claim 1, wherein translating the location of the non-visiblelight dot within the field of view of the camera onto the location onthe display screen includes applying the proportion of said one or moredimensions of the field of view of the camera to correspondingdimensions on the display screen.
 8. The method of claim 1, whereintranslating includes translating the location of the non-visible lightdot onto the location on the display screen irrespective of the size ofthe display screen.
 9. The method of claim 8, wherein the display screenis a television.
 10. The method of claim 1, wherein the non-visiblelight dot is an infrared light dot.
 11. The method of claim 1, whereinprocessing includes searching in the digital video output feed thelocation of the non-visible light dot continually projected by the lighttargeting peripheral unit, and recording the position of the non-visiblelight dot upon actuation of an actuator of the light targetingperipheral unit.
 12. The method of claim 1, wherein processing includessearching the digital video output feed from the camera for thenon-visible light dot that is projected by the light targetingperipheral unit onto the display screen upon actuation of an actuator ofthe light targeting peripheral unit.
 13. The method of claim 1, whereinprocessing the video output feed from the camera with a computerprocessor to identify a location of a non-visible light dot within thefield of view of the camera includes filtering out intensity values ofnon-visible light below a predetermined threshold value.
 14. The methodof claim 1, wherein processing the digital video output feed from thecamera with a computer processor to identify a location of a non-visiblelight dot within the field of view of the camera includes filtering outnon-visible light having a density of pixels below a threshold value.15. A method of operating a video game system, comprising: mechanicallypositioning an optical filter in a field of view of a camera to filterout visible light and allow the camera to view non-visible lightdirected at a display screen positioned within the field of view of thecamera; searching a digital video output feed from the camera with acomputer processor to identify a location of a non-visible light dotwithin the field of view of the camera that is projected onto thedisplay screen by a light targeting peripheral unit; calculating thelocation of the non-visible light dot within the field of view of thecamera as a proportion of one or more dimensions of the field of view ofthe camera; translating the location of the non-visible light dot withinthe field of view of the camera onto a location on the display screen;and initiating a recalibration of the camera to recalculate a boundaryof the display screen in the field of view of the camera if a motion ofthe camera above a threshold amount is sensed with a motion sensor. 16.The method of claim 15, wherein the display screen is a television. 17.The method of claim 15, wherein the non-visible light dot is an infraredlight dot.
 18. The method of claim 15, wherein the non-visible light dotis continually projected by the light targeting peripheral unit onto thedisplay screen and searching includes tracking in the digital videooutput feed the location of the non-visible light dot and recording theposition of the non-visible light dot upon actuation of an actuator ofthe light targeting peripheral unit.
 19. The method of claim 15, whereinthe non-visible light dot is projected by the light targeting peripheralunit onto the display screen only upon actuation of the trigger, andsearching includes searching in the digital video output feed thelocation of the non-visible light dot and recording the position of thenon-visible light dot upon actuation of the actuator of the lighttargeting peripheral unit.
 20. A method of operating a video gamesystem, comprising: positioning an optical filter in a field of view ofa camera to filter out visible light and allow the camera to viewnon-visible light directed at a display screen positioned within thefield of view of the camera; searching a digital video output feed fromthe camera with a computer processor to identify a location within thefield of view of the camera where a non-visible light dot is absent uponactuation of an actuator of a light targeting peripheral that projectsnon-visible light onto the display screen; calculating the locationwithin the field of view of the camera where the non-visible light dotis missing as a proportion of dimensions of the field of view of thecamera; translating the location of the absence of the non-visible lightdot within the field of view of the camera onto a location on thedisplay screen by applying the proportion of said dimensions of thefield of view of the camera to corresponding dimensions on the displayscreen; and initiating a recalibration of the camera to recalculate aboundary of the display screen in the field of view of the camera if amotion of the camera above a threshold amount is sensed with a motionsensor.